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ABSTRACT 
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and the study’s findings are discussed in relation to the North Central 
Association Higher Learning Commission’s ’’Levels of Implementation." (SLD) 
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Obstacles in Outcomes Assessment: 

Identifying and Overcoming Them 

Janet L. Woldt 



Introduction 

It can generally be assumed that anyone attempting to implement an outcomes assessment process - be it student, 
programmatic or institutional - will encounter obstacles that prevent the successful completion of the outcomes assessment process. 
Identifying these obstacles will aid those faculty and administration within institutions of higher education in more effectively 
implementing outcomes assessment programs. This assumption is based upon the premise that successful i mplementation of an 
outcomes assessment program is dependent upon the continuous completion of an outcomes assessment process or cycle. 

This paper will review recent research conducted on obstacles encountered in the outcomes assessment process. The 
particular study highlighted sought to determine: 1) What specific obstacles prevent program administration from successfully 
completing an outcomes assessment process; and, 2) To what degree do these obstacles prevent program administration from 
successfully completing an outcomes assessment process. To answer these two questions, program administration across the nation 
were surveyed. The data revealed the following specific obstacles, which impede progress to a significant degree: insufficient time, 
funding, and personnel to conduct assessment activities; failing support from external audiences, such as alumni and employers of 
graduates, in the outcomes assessment process; the complexity of the outcomes assessment process; institutional personnel’s varying 
levels of understanding of outcomes assessment as a process and a philosophy; and, inadequate communication, leadership and 
guidance concerning assessment matters. Suggestions for solving problems affiliated with these obstacles are also discussed, as are the 
study’s findings in relationship to the North Central Association Higher Learning Commission’s Levels of Implementation. 



Recent Research Conducted 

Using the eight-phase McCann Outcomes Assessment Cycle (Figure 1), the author conducted a study in 2001 identifying 
major obstacles encountered by dental hygiene education program directors in managing programmatic outcomes assessment 
activities. The study of such obstacles is significant because the overwhelming majority of dental hygiene education programs are not 
successfully and effectively completing the outcomes assessment process, as documented by citations contained in accreditation site 
visit reports. 

In the United States, all dental hygiene education programs are sponsored by regionally or nationally accredited public and 
private two- and four-year colleges and universities, with the exception of Alabama. At the program level, outcomes assessment for 
dental hygiene education programs has been mandated by the American Dental Association Commission on Dental Accreditation 
through its Accreditation Standards for Dental Hygiene Education Programs since 1986, when the Commission adopted an outcomes 
assessment standard. Prior to this revision, outcomes assessment was not mandated as a part of dental hygiene education by the 
Commission. Since 1988, when the outcomes assessment standard was implemented, the Commission has observed that those 
schools/programs that most successfully meet this standard treat assessment as an ongoing process that has value apart from 
accreditation (American Dental Association, 2000). 

Because outcomes assessment has been mandated for several years by the Commission on Dental Accreditation, it can be 
assumed that dental hygiene education programs have an outcomes assessment program in place. Therefore, it is not a matter of a 
dental hygiene education program having an outcomes assessment program in place, but of how effective that outcomes assessment 
program is. This level of effectiveness can be determined by the identification of obstacles impeding the outcomes assessment process 
and improved by the removal of those obstacles. 



Methodology 

The target population for this study included the 256 dental hygiene education program directors in the United States who 
represented the 256 dental hygiene education programs accredited by the Commission on Dental Accreditation in 2000. Of that 
number, a sample of 135 dental hygiene education program directors from regionally accredited public and private two- and four-year 
colleges and universities were selected. Of the 135 program directors surveyed, 107 (79%) responded, completing a 65-item self- 
administered survey-type questionnaire, titled A Survey of Dental Hygiene Education Program Directors’ Perceptions of Potential 
Procedural Roadblocks Encountered in Conducting Outcomes Assessment Activities . As noted previously, the McCann Outcomes 
Assessment Cycle (Figure 1), developed by Ann McCann - Director of Assessment, Assessment Center for Health Professions 
Education, Baylor College of Dentistry Texas A&M University - was used extensively in studying the obstacles (i.e., obstacles were 
grouped and analyzed according to the eight phases of the McCann Cycle). 

The survey included five major parts. Part I contained 44 potential procedural obstacles that program administration may 
encounter in conducting outcomes assessment activities. Respondents were asked to rate each of these 44 obstacles on a 5-point 
Likert-type scale as to how much they perceived the obstacle hindered their ability/eflforts to successfully complete the outcomes 
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items. Part HI asked respondents to rate their perceptions on the eight phases of the McCann Outcomes Assessment Cycle using 4- 
point Likert-type scales. Part IV collected demographic data and included: the respondent’s position within the dental hygiene 
education program; who is responsible for conducting assessment activities and for analysis of assessment data; the academic degrees 
offered by the dental hygiene education program; and, the institutional classification. Part V sought to collect data on respondents’ 
past training in outcomes assessment and their perceived training needs. 

Major Obstacles Identified 

Of the 44 obstacles identified for the study, 15 of the obstacles had mean impediment ratings of 3.0 or higher (on a 5-point 
scale) and were rated by greater than 50% of respondents as slightly, moderately or severely impeding their progress in the outcomes 
assessment process. These were identified as the 15 major obstacles and are as follows, with the percentage of respondents in 
parentheses. 



Major Obstacles in Programmatic Outcomes Assessment Ranked According to 
Percent of Surveyed Dental Hygiene Program Directors 
Who Feel That These Obstacles Impede Progress in the Outcomes Assessment Process 



> Inadequate time to spend on assessment activities (90%) 

> Lack of time to follow up on improvements that have been made and implemented as a result of 
outcomes assessment activities (79%) 

> Inadequate number of faculty dedicated to assessment activities (78%) 

> Lack of cooperation from employers of graduates of the program in conducting assessment activities, 
e.g., completing surveys (78%) 

> Faculty’s lack of preparedness/training in outcomes assessment activities (75%) 

> Lack of financial resources to follow up on improvements (74%) 

> Complexity of the outcomes assessment process (74%) 

^ Lack of cooperation from alumni of the program in conducting assessment activities, eg., 
completing surveys (73%) 

> Inadequate number of clerical staff available to assist with assessment activities (71%) 

> Inadequate availability of training in outcomes assessment (67%) 

> Lack of a departmental committee that is specifically responsible for assessment activities (67%) 

> Inadequate funding for assessment activities (63%) 

> Lack of communication with and isolation from other departments within the institution regarding 
assessment activities, e g., campus-wide assessment committee (61%) 

> Lack of a dental hygiene assessment committee that provides guidance on using assessment 
information for program improvement (60%) 

> Lack of knowledgeable staff to assist in conducting data analysis (56%) 

These 15 major obstacles can be grouped into six separate categories. 

1) Lack of time to conduct assessment activities; 

2) Lack of funding to conduct assessment activities; 

3) Lack of faculty/staff to conduct or assist in assessment activities; 

4) Lack of cooperation from external audiences; 

5) Complexity of the outcomes assessment process hindered by the unpreparedness of 
faculty, who do not have access to appropriate training in assessment; and, 

6) Lack of communication and guidance on outcomes assessment from within the 
program/department and from within the institution. 



Generalizability of the Study’s Findings 

Because the results of this study encompass universal concerns among the whole of higher education (encompassing multiple 
types of institutions), the results can be applied not only to programmatic outcomes assessment but to student and institutional 
outcomes assessment as well. Therefore, even though this study was conducted on programmatic outcomes assessment, faculty, staff, 
and/or administration involved in implementing an outcomes assessment process for students, programs or institutions may benefit by 
closely examining the results of this data. 
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Overcoming Obstacles 

Overcoming obstacles in outcomes assessment is largely dependent on institutional personnel’s (who manage outcomes 
assessment programs) knowledge and awareness of appropriate outcomes assessment-related support and resources. The first four 
categories of major obstacles - lack of time, funding, personnel, and cooperation from external audiences in conducting outcomes 
assessment - concern appropriate support and are common concerns among the whole of higher education. Such common or “basic 
needs” concerns are addressed in the following practical strategies to overcome outcomes assessment obstacles (adapted from 
Concordia College, 2000). 



Practical Strategies to Overcome Obstacles Encountered in Outcomes Assessment 



^ Assessment should be an ongoing process. Do not assess everything at the same time. Spread assessment activities out 
over the academic year and summer. 

^ Work as a team with your department. Do not have one person do all of the work. 

^ Cooperate with other departments. Faculty from other departments could facilitate focus groups. 

^ Do as much as possible within the context of things you are already doing/getting paid to do. 

^ Borrow methods and instruments from other departments or other institutions. 

■ Saves time - no need to "recreate the wheel." 

■ Find out how well it worked for the others. Learn from other’s success and mistakes. 

■ Attend institutional assessment meetings or symposia to hear what other departments are doing. 

■ Tap into assessment newsletters or web links to assessment at other institutions. 

^ Buy existing instruments, where appropriate, instead of creating them. 

■ Saves time. 

■ May provide comparative information. 

^ Make instruments, assignments or samples as short as possible, yet still provide pertinent information. 

■ This encourages optimal use of student time, faculty time, and analysis time. 

■ Focus on your goals. Do not ask questions about unrelated things or collect information that will not provide 
feedback about the goals. 

■ On surveys, do not ask several questions about the same thing. 

^ Delegate assessment tasks when possible. Enlist the help of 

■ Support staff in your department. 

■ Student workers (eg., data entry). 

■ Students enrolled in independent study (eg., working with design of instruments or analysis of results), when this 
matches what students are studying. 

^ Use electronic media as much as possible. 

■ Database and spreadsheet applications greatly simplify analysis of data. 

■ Electronic storage of portfolios, papers, etc . . . 

^ Take advantage of resources available on campus, such as 

■ Workshops and discussion sessions. 

* Consultation with department or institutional assessment committee. 

8 Institutional Research or Institutional Effectiveness office. 

^ Use tried and tested assessment tools, such as 

B Cross and Angelo’s Teaching Goals Inventory (1993) for student and course assessment, to make the outcomes 
assessment process more efficient. 

H A matrix, as suggested by the NCA Higher Learning Commission, which forces the clarification of goals and the 
development of processes to objectively determine how, when and if the goals are being met. 
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Complexity of Outcomes Assessment: Levels of Understanding 

The last two categories of major obstacles concern the complexity of the outcomes assessment process, the lack of 
knowledge about outcomes assessment among those expected to implement an outcomes assessment program, and the lack of 
communication and guidance from mid- and upper-level administration concerning outcomes assessment. 

Among faculty and administration at institutions of higher education, there are many different levels of understanding about 
outcomes assessment as a philosophy and as a process. The results of the author’s study indicated the levels at which programs are 
engaged in the outcomes assessment process vary from vague to intimate familiarity with the process. Some institutional personnel are 
able to see the “big picture” of assessment and the potential of the outcomes process to improve teaching and student learning. Others 
see assessment only at the student level, but not as affecting the institution as a whole. 

These varying levels must be recognized and addressed by an institution’s administration. A means by which this could occur 
are institutionally sponsored workshops on outcomes assessment offered at the beginner, intermediate and advanced levels of 
understanding. Further, any printed materials that the institution publishes must also address the levels of understanding as well. All 
faculty, staff and administration within an institution must be provided with a solid foundation in the rudimentary aspects of outcomes 
assessment before effective and successful implementation of an outcomes assessment process can occur. 



Effective Communication 

Another aspect concerning the complexity of outcomes assessment is the perceived uncertainty of the accreditation guidelines 
regarding assessment. At many institutions, there is confusion as to what exactly must be provided to an accrediting agency to satisfy 
the requirements. The NCA Higher Learning Commission provides detailed materials, and sponsors conferences and workshops to 
guide institutions in outcomes assessment; however, this information must be processed and unanimously agreed upon by upper-level 
administration. Therefore, institutional support of outcomes assessment must be seamlessly and accordingly relayed from upper-level 
administration to mid-level administration to faculty. 

In order for institutional administration to enjoy the successful implementation of an outcomes assessment process, upper- 
level administration must relay consistent and clear information to institutional personnel. The communication of accurate information 
regarding outcomes assessment is key in implementing an outcomes assessment process. This communication is largely dependent 
upon the leadership and guidance provided by the administration. 



Leadership and Guidance 

Because of the complexity of the outcomes assessment process, faculty and administration may generally find the process 
difficult to understand and difficult to implement. Further, the complex process of assessment has been added to the already 
challenging and taxing jobs of faculty and administration. Generally speaking, the way in which administration chooses to implement 
the decision-making process plays a critical role in faculty’s reaction to and support of decisions made concerning outcomes 
assessment. Leadership and guidance in this process must be consistently provided by the administration throughout the institution to 
ensure the successful implementation of an outcomes assessment process. 

Good leaders are those people who can mobilize human, material, and symbolic resources toward specific ends (Curry, 1992, 
p. 20). Institutional leadership must effectively facilitate the outcomes assessment process, through performing tasks such as gathering 
information, communicating with other members of the institution, developing new coalitions, and identifying existing coalitions. It is 
imperative that upper administration convey to the faculty and mid-level administration that they are integral stakeholders in the 
outcomes assessment process (1992). Effective leadership must also be able to communicate that the results of outcomes assessment 
are not immediate. It may take several months to a few years to see results and to, “...permeate all aspects of campus culture with 
structures that make assessment self-sustaining” (Gray, 1997, p. 13). 



Relating the Study’s Findings to the NCA Levels of Implementation 

In 2000, the North Central Association Higher Learning Commission (NC A-HLC) developed Levels of Implementation to 
clarify and objectify its stance on outcomes assessment. 



These Levels are presented in the form of a matrix and consist of three levels of implementation and four patterns of 
characteristics or descriptors associated with each level. The patterns, associated with Level One, “Beginning 
Implementation of Assessment Programs,” include a number of characteristics consistent with assessment efforts 
that are in their infancy, or that are progressing at a slower than desired pace or that have stalled. Patterns associated 
with Level Two, “Making Progress in Implementing Assessment Programs,” include characteristics consistent with 
the value the institution, its academic departments, and each of its academic programs place on measuring student 
learning and assessing the outcomes against clearly specified goals and measurable objectives and outcomes in the 
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cognitive, behavioral, and affective domains. Patterns associated with Level Three, “Maturing Stages of Continuous 
Improvement,” include characteristics that have been culled from those assessment programs that are structured, 
systematic, on going, and sustainable. In institutions that manifest this level of attainment in their assessment 
programs, assessment has become a way of life (Lopez, 2000, p. 2). 



When institutional administration and faculty can identify obstacles that hinder progress, they will be better able to develop 
solutions to overcome those obstacles and achieve the upper Levels of Implementation. The fewer obstacles that are present, the more 
amenable administration and faculty will be to work with/continue to work with outcomes assessment programs. Further, with the 
knowledge of the NCA-HLC Levels, potential obstacles, and solutions administration and faculty will be sufficiently equipped to 
work toward developing an all-encompassing culture of assessment at their institution. 

As noted by Lopez (2000), “The Commission expects that the document. Levels of Implementation, will be a tool that our 
affiliated institutions can use to understand and strengthen their assessment programs.” So too can the results of the aforementioned 
research on obstacles be used to strengthen student, programmatic and institutional outcomes assessment programs. 



Conclusion 

The terms and processes associated with outcomes assessment in higher education are not new and have been in existence for 
over 20 years. However, outcomes assessment is still perceived as new by many faculty and administration at the grass roots level. 
This newness may be attributed to the near exclusive discussion of outcomes assessment at the “ivory tower” upper administration 
level, and not at the grass roots level. Further, outcomes assessment models and processes have been classified by seasoned experts in 
the field as complex and difficult to understand (Gray, 1997). Because of its perceived newness and complexity, faculty and 
administration are at different levels in understanding what assessment is and how it works. Acknowledgement of these various levels 
is key as institutional administration formulates and develops plans to implement an outcomes assessment program. 

The obstacles, frustrations and struggles experienced by faculty and administration regarding the complexity of outcomes 
assessment and effective leadership and guidance in the implementation of an outcomes assessment program are common among 
institutions of higher education in the United States. Messina and Fagans state that this is a nationwide concern because, “All 
institutions of higher education are subject to regulations and policies from outside the institution...” (1992, p. 8). Many faculty are 
first exposed to outcomes assessment when their institution is in the process of preparing a self-study for an external accrediting 
agency. When outcomes assessment is introduced on this personal level (i.e., faculty involved in preparing the self-study) faculty 
begin to understand what outcomes assessment really means (Gray, 1997). However, the true benefit of outcomes assessment is 
realized when those implementing it acknowledge its use as not just an accreditation requirement but as a means to bolster student 
academic achievement through the improvement of teaching and student learning. 
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Figure 1. The McCann Outcomes Assessment Cycle 
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